Gaze detection in a 3D mapping environment

ABSTRACT

A method, including receiving a three-dimensional (3D) map of at least a part of a body of a user (22) of a computerized system, and receiving a two dimensional (2D) image of the user, the image including an eye (34) of the user. 3D coordinates of a head (32) of the user are extracted from the 3D map and the 2D image, and a direction of a gaze performed by the user is identified based on the 3D coordinates of the head and the image of the eye.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/256,654, filed Sep. 5, 2016, which is a continuation of U.S. patentapplication Ser. No. 13/960,822, filed Aug. 7, 2013 (now U.S. Pat. No.9,454,225), which is a continuation of PCT Patent ApplicationPCT/IB2012/050577, filed Feb. 9, 2012, which claims the benefit of U.S.Provisional Patent Application 61/440,877, filed on Feb. 9, 2011, U.S.Provisional Patent Application 61/526,692, filed on Aug. 24, 2011, andU.S. Provisional Patent Application 61/538,867, filed on Sep. 25, 2011,which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to human-machine interfaces, andspecifically to interfaces that combine multiple user interactionmodalities.

BACKGROUND

Many different types of user interface devices and methods are currentlyavailable. Common tactile interface devices include a computer keyboard,a mouse and a joystick. Touch screens detect the presence and locationof a touch by a finger or other object within the display area. Infraredremote controls are widely used, and “wearable” hardware devices havebeen developed, as well, for purposes of remote control.

Computer interfaces based on three-dimensional (3D) sensing of parts ofa user's body have also been proposed. For example, PCT InternationalPublication WO 03/071410, whose disclosure is incorporated herein byreference, describes a gesture recognition system using depth-perceptivesensors. A 3D sensor, typically positioned in a room in proximity to theuser, provides position information, which is used to identify gesturescreated by a body part of interest. The gestures are recognized based onthe shape of the body part and its position and orientation over aninterval. The gesture is classified for determining an input into arelated electronic device.

Documents incorporated by reference in the present patent applicationare to be considered an integral part of the application except that tothe extent any terms are defined in these incorporated documents in amanner that conflicts with the definitions made explicitly or implicitlyin the present specification, only the definitions in the presentspecification should be considered.

As another example, U.S. Pat. No. 7,348,963, whose disclosure isincorporated herein by reference, describes an interactive video displaysystem, in which a display screen displays a visual image, and a cameracaptures 3D information regarding an object in an interactive arealocated in front of the display screen. A computer system directs thedisplay screen to change the visual image in response to changes in theobject.

Three-dimensional human interface systems may identify not only theuser's hands, but also other parts of the body, including the head,torso and limbs. For example, U.S. Patent Application Publication2010/0034457, whose disclosure is incorporated herein by reference,describes a method for modeling humanoid forms from depth maps. Thedepth map is segmented so as to find a contour of the body. The contouris processed in order to identify a torso and one or more limbs of thesubject. An input is generated to control an application program runningon a computer by analyzing a disposition of at least one of theidentified limbs in the depth map.

Some user interface systems track the direction of the user's gaze. Forexample, U.S. Pat. No. 7,762,665, whose disclosure is incorporatedherein by reference, describes a method of modulating operation of adevice, comprising: providing an attentive user interface for obtaininginformation about an attentive state of a user; and modulating operationof a device on the basis of the obtained information, wherein theoperation that is modulated is initiated by the device. Preferably, theinformation about the user's attentive state is eye contact of the userwith the device that is sensed by the attentive user interface.

SUMMARY

There is provided, in accordance with an embodiment of the presentinvention a method including receiving a three-dimensional (3D) map ofat least a part of a body of a user of a computerized system, receivinga two dimensional (2D) image of the user, the image including an eye ofthe user, extracting, from the 3D map and the 2D image, 3D coordinatesof a head of the user, and identifying, based on the 3D coordinates ofthe head and the image of the eye, a direction of a gaze performed bythe user.

There is also provided, in accordance with an embodiment of the presentinvention a method including receiving an image including an eye of auser of a computerized system, identifying, based the image of the eye,a direction of a gaze performed by the user, identifying, based on thedirection of the gaze, a region on a display coupled to the computerizedsystem, and performing an operation on content presented in the region.

There is additionally provided, in accordance with an embodiment of thepresent invention a method including presenting, by a computerizedsystem, multiple interactive items on a display coupled to the computer,receiving, from a sensing device coupled to the computer, an inputrepresenting a gaze direction of a user, identifying a target point onthe display based on the gaze direction, associating the target pointwith a first interactive item appearing on the display, and responsivelyto the target point, opening one or more second interactive items on thedisplay.

There is further provided, in accordance with an embodiment of thepresent invention a method including receiving and segmenting a firstsequence of three-dimensional (3D) maps over time of at least a part ofa body of a user of a computerized system in order to extract 3Dcoordinates of a first point and a second point of the user, the 3D mapsindicating a motion of the second point with respect to a displaycoupled to the computerized system, calculating a line segment thatintersects the first point and the second point, identifying a targetpoint where the line segment intersects the display, and engaging aninteractive item presented on the display in proximity to the targetpoint.

There is also provided, in accordance with an embodiment of the presentinvention an apparatus including a sensing device configured to receivea three dimensional (3D) map of at least a part of a body of a user andan image of an eye of the user, and to receive a two dimensional (2D)image of the user, the 2D image including an eye of the user, and acomputer coupled to the sensing device and configured to extract, fromthe 3D map and the 2D image, 3D coordinates of a head of the user and toidentify, based on the 3D coordinates of the head and the image of theeye, a direction of a gaze performed by the user.

There is additionally provided, in accordance with an embodiment of thepresent invention an apparatus including a sensing device configured toreceive an image including an eye of a user, and a computer configuredto identify, based the image of the eye, a direction of a gaze performedby the user, to identify, based on the direction of the gaze, a regionon a display coupled to the computerized system, and to perform anoperation on content presented in the region.

There is further provided, in accordance with an embodiment of thepresent invention an apparatus including a display, and a computercoupled to the display and configured to present multiple interactiveitems on the display, to receive, from a sensing device coupled to thecomputer, an input representing a gaze direction of a user, to identifya target point on the display based on the gaze direction, to associatethe target point with a first interactive item appearing on the display,and responsively to the target point, to open one or more secondinteractive items on the display.

There is also provided, in accordance with an embodiment of the presentinvention an apparatus including a display, and a computer coupled tothe display and configured to receive and segment a first sequence ofthree-dimensional (3D) maps over time of at least a part of a body of auser of a computerized system in order to extract 3D coordinates of afirst point and a second point of the user, the 3D maps indicating amotion of the second point with respect to a display coupled to thecomputer system, to calculate a line segment that intersects the firstpoint and the second point, to identify a target point where the linesegment intersects the display, and to engage an interactive itempresented on the display in proximity to the target point.

There is additionally provided, in accordance with an embodiment of thepresent invention a computer software product including a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer, cause the computer toreceive a three-dimensional (3D) map of at least a part of a body of auser of the computer, to receive a two dimensional (2D) image of theuser, the image including an eye of the user, to extract, from the 3Dmap and the 2D image, 3D coordinates of a head of the user, and toidentify, based on the 3D coordinates of the head and the image of theeye, a direction of a gaze performed by the user.

There is further provided, in accordance with an embodiment of thepresent invention a computer software product including a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer, cause the computer toreceive an image including an eye of a user of the computer system, toidentify, based the image of the eye, a direction of a gaze performed bythe user, to identify, based on the direction of the gaze, a region on adisplay coupled to the computerized system, and to perform an operationon content presented in the region.

There is also provided, in accordance with an embodiment of the presentinvention a computer software product including a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer, cause the computer topresent multiple interactive items on a display coupled to the computer,to receive, from a sensing device coupled to the computer, an inputrepresenting a gaze direction of a user, to identify a target point onthe display based on the gaze direction, to associate the target pointwith a first interactive item appearing on the display, and responsivelyto the target point, to open one or more second interactive items on thedisplay.

There is additionally provided, in accordance with an embodiment of thepresent invention a computer software product including a non-transitorycomputer-readable medium, in which program instructions are stored,which instructions, when read by a computer, cause the computer toreceive and segment a first sequence of three-dimensional (3D) maps overtime of at least a part of a body of a user of the computer in order toextract 3D coordinates of a first point and a second point of the user,the 3D maps indicating a motion of the second point with respect to adisplay coupled to the computer, to calculate a line segment thatintersects the first point and the second point, to identify a targetpoint where the line segment intersects the display, and to engage aninteractive item presented on the display in proximity to the targetpoint.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIG. 1 is a schematic, pictorial illustration of a computer systemimplementing a mixed-modality user interface, in accordance with anembodiment of the present invention;

FIG. 2 is a block diagram that schematically illustrates functionalcomponents of the computer system implementing the mixed-modality userinterface, in accordance with an embodiment of the present invention;

FIG. 3 is a flow diagram that schematically illustrates a method ofdetecting a direction of a gaze of a user, in accordance with anembodiment of the present invention;

FIG. 4 is a schematic representation of a numeric keypad configured forentering a password, in accordance with an embodiment of the presentinvention;

FIG. 5 is a flow diagram that schematically illustrates a method ofinteracting with a gaze operated user interface, in accordance with anembodiment of the present invention;

FIGS. 6A-6C are schematic representations illustrating a sequence ofoperations performed using the gaze operated user interface, inaccordance with an embodiment of the present invention;

FIG. 7 is a flow diagram that schematically illustrates a method ofdetecting a gaze related pointing gesture, in accordance with anembodiment of the present invention;

FIG. 8 is a schematic pictorial illustration of the user performing aPoint-Select gesture to select a first given interactive item, inaccordance with an embodiment of the present invention;

FIG. 9 is a schematic pictorial illustration of the user performing aPoint-Touch gesture to manipulate a second given interactive item, inaccordance with an embodiment of the present invention;

FIG. 10 is a schematic pictorial illustration showing an alternativePoint-Select gesture, also referred to herein as a Trigger gesture, inaccordance with an embodiment of the present invention;

FIG. 11 is a schematic pictorial illustration of the user pointing ahand at a given icon presented on a display, in order to calibrate thecomputer system, in accordance with an embodiment of the presentinvention; and

FIGS. 12A and 12B are schematic pictorial illustrations of the computersystem assisting the user to select a given icon by presenting the iconsin smaller and larger sizes, in accordance with an embodiment of thepresent invention.

DETAILED DESCRIPTION OF EMBODIMENTS Overview

When using physical tactile input devices such as buttons, rollers ortouch screens, a user typically engages and disengages control of a userinterface by touching and/or manipulating the physical device.Embodiments of the present invention describe pointing gestures forengaging interactive items presented on a display coupled to a computerexecuting a mixed modality user interface that includesthree-dimensional (3D) sensing, by a 3D sensor, of motion or change ofposition of one or more body parts, typically a hand or a finger, of theuser. Pointing gestures described herein include Point-Select,Point-Touch and Point-Hold gestures that are explained in detailhereinbelow.

The Point-Select gesture enables the user to select an interactive itempresented on the display. For example, using the Point-Select gesture,the user can start watching a movie by performing the Point-Selectgesture toward an icon (on the display) associated with the move. ThePoint-Touch gesture enables the user to manipulate an interactive itempresented on the display. For example, the user can horizontally scrolla list of interactive items (e.g., movies) presented on the display bymanipulating a horizontal scroll box via the Point-Touch gesture. ThePoint-Hold gesture enables the user to view context information for aninteractive item presented on the display. For example, in response tothe user performing a Point-Hold gesture on an icon representing amovie, the computer can present a pop-up window including informationsuch as a plot summary, a review, and cast members. In some embodiments,the mixed modality user interface may also convey visual feedback as theuser performs the pointing gestures described supra.

While interacting with traditional two-dimensional (2D) user interfaces,the physical devices described supra typically convey tactile feedbackto the user. However, while interacting with a 3D user interface such asthe mixed modality user interface described herein, the user may performgestures without engaging any physical device, and therefore not receiveany tactile feedback. Embodiments of the present invention providemethods and systems for interacting with items presented on a display,and receiving non-tactile feedback, thereby compensating for the lack oftactile feedback.

System Description

FIG. 1 is a schematic, pictorial illustration of a mixed-modality userinterface 20 for operation by a user 22 of a computer 26, in accordancewith an embodiment of the present invention. (Although for the sake ofsimplicity, only a single user and user interface are shown in thefigure, in practice interface 20 may interact with multiple usersconcurrently. Alternative embodiments of the present invention may usedifferent user interfaces and/or support multiple user interfaces acrossdifferent devices). User interface 20 in the pictured embodiment isbased, by way of example, on a 3D sensing device 24, which captures 3Dscene information that includes a body, or at least parts of the body,such as a finger 30, a hand 31, a head 32, or eyes 34. Device 24 or aseparate camera (not shown in the figures) may also capture color videoimages of the scene. The information captured by device 24 is processedby computer 26, which drives a display screen 28 accordingly to presentand manipulate on-screen interactive items 36 (also referred to hereinas interactive items). Alternatively, the user interface may be used inconjunction with any type of computerized equipment, such as a laptop, atablet computer, a television, etc.

While FIG. 1 shows computer 26 in a tower configuration, otherconfigurations of the computer are considered to be within the spiritand scope of the present invention. For example, computer 26 may beconfigured as a desktop computer, a portable computer (e.g., a laptop)or an all-in-one computer.

Computer 26 processes data generated by device 24 in order toreconstruct a 3D map of user 22. The term “3D map” (or equivalently,“depth map”) refers to a set of 3D coordinates representing a surface ofa given object, in this case the user's body. In one embodiment, device24 projects a pattern of spots onto the object and captures an image ofthe projected pattern. Computer 26 then computes the 3D coordinates ofpoints on the surface of the user's body by triangulation, based ontransverse shifts of the spots in the imaged pattern. The 3D coordinatesare measured, by way of example, with reference to a generallyhorizontal X-axis 40, a generally vertical Y-axis 42 and a depth Z-axis44, based on device 24. Methods and devices for this sort oftriangulation-based 3D mapping using a projected pattern are described,for example, in PCT International Publications WO 2007/043036, WO2007/105205 and WO 2008/120217, whose disclosures are incorporatedherein by reference. Alternatively, system 20 may use other methods of3D mapping, using single or multiple cameras or other types of sensors,as are known in the art.

In some embodiments, device 24 detects the location and direction ofeyes 34 of user 22, typically by processing and analyzing an imagecomprising light (typically infrared and/or a color produced by thered-green-blue additive color model) reflecting from one or both eyes34, in order to find a direction of the user's gaze. In alternativeembodiments, computer 26 (either by itself or in combination with device24) detects the location and direction of the eyes 34 of the user. Thereflected light may originate from a light projecting source of device24, or any other natural (e.g., sunlight) or artificial (e.g., a lamp)source. Using techniques that are known in the art such as detectingpupil center and corneal reflections (PCCR), device 24 may process andanalyze an image comprising light reflecting from an element of eye 34,such as a pupil 38, an iris 39 or a cornea 41, in order to find thedirection of the user's gaze. Additionally, device 24 may convey (tocomputer 26) the light reflecting from the cornea as a glint effect.

The location and features of the user's head (e.g., an edge of the eye,a nose or a nostril) that are extracted by computer 26 from the 3D mapmay be used in finding coarse location coordinates of the user's eyes,thus simplifying the determination of precise eye position and gazedirection, and making the gaze measurement more reliable and robust.Furthermore, computer 26 can readily combine the 3D location of parts ofhead 32 (e.g., eye 34) that are provided by the 3D map with gaze angleinformation obtained via eye part image analysis in order to identify agiven on-screen object 36 at which the user is looking at any giventime. This use of 3D mapping in conjunction with gaze tracking allowsuser 22 to move head 32 freely while alleviating the need to activelytrack the head using sensors or emitters on the head, as in some eyetracking systems that are known in the art.

By tracking eye 34, embodiments of the present invention may reduce theneed to re-calibrate user 22 after the user moves head 32. In someembodiments, computer 26 may use depth information for head 32, eye 34and pupil 38, in order to track the head's movement, thereby enabling areliable gaze angle to be calculated based on a single calibration ofuser 22. Utilizing techniques that are known in the art such as PCCR,pupil tracking, and pupil shape, computer 26 may calculate a gaze angleof eye 34 from a fixed point of head 32, and use the head's locationinformation in order to re-calculate the gaze angle and enhance theaccuracy of the aforementioned techniques. In addition to reducedrecalibrations, further benefits of tracking the head may includereducing the number of light projecting sources and reducing the numberof cameras used to track eye 34.

Computer 26 typically comprises a general-purpose computer processor,which is programmed in software to carry out the functions describedhereinbelow. The software may be downloaded to the processor inelectronic form, over a network, for example, or it may alternatively beprovided on non-transitory tangible computer-readable media, such asoptical, magnetic, or electronic memory media. Alternatively oradditionally, some or all of the functions of the computer processor maybe implemented in dedicated hardware, such as a custom or semi-customintegrated circuit or a programmable digital signal processor (DSP).Although computer 26 is shown in FIG. 1, by way of example, as aseparate unit from sensing device 24, some or all of the processingfunctions of the computer may be performed by suitable dedicatedcircuitry within the housing of the sensing device or otherwiseassociated with the sensing device.

As another alternative, these processing functions may be carried out bya suitable processor that is integrated with display 28 (in a televisionset, for example) or with any other suitable sort of computerizeddevice, such as a game console or a media player. The sensing functionsof device 24 may likewise be integrated into the computer or othercomputerized apparatus that is to be controlled by the sensor output.

Various techniques may be used to reconstruct the 3D map of the body ofuser 22. In one embodiment, computer 26 extracts 3D connected componentscorresponding to the parts of the body from the depth data generated bydevice 24. Techniques that may be used for this purpose are described,for example, in U.S. patent application Ser. No. 12/854,187, filed Aug.11, 2010, whose disclosure is incorporated herein by reference. Thecomputer analyzes these extracted components in order to reconstruct a“skeleton” of the user's body, as described in the above-mentioned U.S.Patent Application Publication 2010/0034457, or in U.S. patentapplication Ser. No. 12/854,188, filed Aug. 11, 2010, whose disclosureis also incorporated herein by reference. In alternative embodiments,other techniques may be used to identify certain parts of the user'sbody, and there is no need for the entire body to be visible to device24 or for the skeleton to be reconstructed, in whole or even in part.

Using the reconstructed skeleton, computer 26 can assume a position of abody part such as a tip of finger 30, even though the body part (e.g.,the fingertip) may not be detected by the depth map due to issues suchas minimal object size and reduced resolution at greater distances fromdevice 24. In some embodiments, computer 26 can auto-complete a bodypart based on an expected shape of the human part from an earlierdetection of the body part, or from tracking the body part along several(previously) received depth maps.

In some embodiments, the information generated by computer 26 as aresult of this skeleton reconstruction includes the location anddirection of the user's head, as well as of the arms, torso, andpossibly legs, hands and other features, as well. Changes in thesefeatures from frame to frame (i.e. depth maps) or in postures of theuser can provide an indication of gestures and other motions made by theuser. User posture, gestures and other motions may provide a controlinput for user interaction with interface 20. These body motions may becombined with other interaction modalities that are sensed by device 24,including user eye movements, as described above, as well as voicecommands and other sounds. Interface 20 thus enables user 22 to performvarious remote control functions and to interact with applications,interfaces, video programs, images, games and other multimedia contentappearing on display 28.

FIG. 2 is a block diagram that schematically illustrates functionalcomponents of user interface 20, in accordance with an embodiment of thepresent invention. Sensing device 24 comprises an illuminationsubassembly 50, which projects a pattern onto the scene of interest. Adepth imaging subassembly 52, such as a suitably-configured videocamera, captures images of the pattern on the scene. Typically,illumination subassembly 50 and imaging subassembly 52 operate in theinfrared range, although other spectral ranges may also be used.Optionally, a color video camera (not shown) in device 24 captures 2Dcolor images of the scene, and a microphone 54 may also capture sound.

A processor 56 receives the images from subassembly 52 and compares thepattern in each image to a reference pattern stored in a memory 58. Thereference pattern is typically captured in advance by projecting thepattern onto a reference plane at a known distance from device 24.Processor 56 computes local shifts of parts of the pattern over the areaof the 3D map and translates these shifts into depth coordinates.Details of this process are described, for example, in PCT InternationalPublication WO 2010/004542, whose disclosure is incorporated herein byreference. Alternatively, as noted earlier, device 24 may be configuredto generate 3D maps by other means that are known in the art, such asstereoscopic imaging, sonar-like devices (sound based/acoustic),wearable implements, lasers, or time-of-flight measurements.

Processor 56 typically comprises an embedded microprocessor, which isprogrammed in software (or firmware) to carry out the processingfunctions that are described hereinbelow. The software may be providedto the processor in electronic form, over a network, for example;alternatively or additionally, the software may be stored onnon-transitory tangible computer-readable media, such as optical,magnetic, or electronic memory media. Processor 56 also comprisessuitable input and output interfaces and may comprise dedicated and/orprogrammable hardware logic circuits for carrying out some or all of itsfunctions. Details of some of these processing functions and circuitsthat may be used to carry them out are presented in the above-mentionedPublication WO 2010/004542.

In some embodiments, a gaze sensor 60 detects the gaze direction of eyes34 of user 22 by capturing and processing two dimensional images of user22. In alternative embodiments, computer 26 detects the gaze directionby processing a sequence of 3D maps conveyed by device 24. Sensor 60 mayuse any suitable method of eye tracking that is known in the art, suchas the method described in the above-mentioned U.S. Pat. No. 7,762,665or in U.S. Pat. No. 7,809,160, whose disclosure is incorporated hereinby reference, or the alternative methods described in references citedin these patents. For example, sensor 60 may capture an image of light(typically infrared light) that is reflected from the fundus and/or thecornea of the user's eye or eyes. This light may be projected toward theeyes by illumination subassembly 50 or by another projection element(not shown) that is associated with sensor 60. Sensor 60 may capture itsimage with high resolution over the entire region of interest of userinterface 20 and may then locate the reflections from the eye withinthis region of interest. Alternatively, imaging subassembly 52 maycapture the reflections from the user's eyes (ambient light, reflectionfrom monitor) in addition to capturing the pattern images for 3Dmapping.

As another alternative, processor 56 may drive a scan control 62 todirect the field of view of gaze sensor 60 toward the location of theuser's face or eye 34. This location may be determined by processor 60or by computer 26 on the basis of a depth map or on the basis of theskeleton reconstructed from the 3D map, as described above, or usingmethods of image-based face recognition that are known in the art. Scancontrol 62 may comprise, for example, an electromechanical gimbal, or ascanning optical or optoelectronic element, or any other suitable typeof scanner that is known in the art, such as a microelectromechanicalsystem (MEMS) based mirror that is configured to reflect the scene togaze sensor 60.

In some embodiments, scan control 62 may also comprise an optical orelectronic zoom, which adjusts the magnification of sensor 60 dependingon the distance from device 24 to the user's head, as provided by the 3Dmap. The above techniques, implemented by scan control 62, enable a gazesensor 60 of only moderate resolution to capture images of the user'seyes with high precision, and thus give precise gaze directioninformation.

In alternative embodiments, computer 26 may calculate the gaze angleusing an angle (i.e., relative to Z-axis 44) of the scan control. Inadditional embodiments, computer 26 may compare scenery captured by thegaze sensor 60, and scenery identified in 3D depth maps. In furtherembodiments, computer 26 may compare scenery captured by the gaze sensor60 with scenery captured by a 2D camera having a wide field of view thatincludes the entire scene of interest. Additionally or alternatively,scan control 62 may comprise sensors (typically either optical orelectrical) configured to verify an angle of the eye movement.

Processor 56 processes the images captured by gaze sensor 60 in order toextract the user's gaze angle. By combining the angular measurementsmade by sensor 60 with the 3D location of the user's head provided bydepth imaging subassembly 52, the processor is able to derive accuratelythe user's true line of sight in 3D space. The combination of 3D mappingwith gaze direction sensing reduces or eliminates the need for precisecalibration and comparing multiple reflection signals in order toextract the true gaze direction. The line-of-sight information extractedby processor 56 enables computer 26 to identify reliably the interactiveitem at which the user is looking.

The combination of the two modalities can allow gaze detection withoutusing an active projecting device (i.e., illumination subassembly 50)since there is no need for detecting a glint point (as used, forexample, in the PCCR method). Using the combination can solve theglasses reflection known problem of other gaze methods that are known inthe art. Using information derived from natural light reflection, the 2Dimage (i.e. to detect the pupil position), and the 3D depth map (i.e.,to identify the head's position by detecting the head's features),computer 26 can calculate the gaze angle and identify a giveninteractive item 36 at which the user is looking.

As noted earlier, gaze sensor 60 and processor 56 may track either oneor both of the user's eyes. If both eyes 34 are tracked with sufficientaccuracy, the processor may be able to provide an individual gaze anglemeasurement for each of the eyes. When the eyes are looking at a distantobject, the gaze angles of both eyes will be parallel; but for nearbyobjects, the gaze angles will typically converge on a point in proximityto an object of interest. This point may be used, together with depthinformation, in extracting 3D coordinates of the point on which theuser's gaze is fixed at any given moment.

As mentioned above, device 24 may create 3D maps of multiple users whoare in its field of view at the same time. Gaze sensor 60 may similarlyfind the gaze direction of each of these users, either by providing asingle high-resolution image of the entire field of view, or by scanningof scan control 62 to the location of the head of each user.

Processor 56 outputs the 3D maps and gaze information via acommunication link 64, such as a Universal Serial Bus (USB) connection,to a suitable interface 66 of computer 26. The computer comprises acentral processing unit (CPU) 68 with a memory 70 and a user interface72, which drives display 28 and may include other components, as well.As noted above, device 24 may alternatively output only raw images, andthe 3D map and gaze computations described above may be performed insoftware by CPU 68. Middleware for extracting higher-level informationfrom the 3D maps and gaze information may run on processor 56, CPU 68,or both. CPU 68 runs one or more application programs, which drive userinterface 72 based on information provided by the middleware, typicallyvia an application program interface (API). Such applications mayinclude, for example, games, entertainment, Web surfing, and/or officeapplications.

Although processor 56 and CPU 68 are shown in FIG. 2 as separatefunctional elements with a certain division of processing tasks betweenthem, the functions of the processor and CPU may alternatively becarried out by a single processing unit, or these functions may bedivided among three or more processing units. Furthermore, althoughdevice 24 is shown as containing a certain combination of components ina particular arrangement, other device configurations may be used forthe purposes described herein, and are considered to be within the scopeof the present invention.

Gaze Detection

FIG. 3 is a flow diagram that schematically illustrates a method ofdetecting a direction of a gaze of user 22, in accordance with anembodiment of the present invention. In an initial step 80, computer 26receives, from depth imaging subassembly 52, a 3D map including at leasta body part of user 22, and in an extraction step 82, the computersegments the received 3D map in order to extract 3D coordinates of head32. In a receive step 84, computer 26 receives, from gaze sensor 60, atwo dimensional image of the user including eye 34. As described supra,illumination subassembly 50 may project a light toward user 22, and thereceived image may comprise light reflected off the fundus and/or thecornea of eye(s) 34. In an identification step 86, computer 26 analyzesthe received depth map and image in order to identify a gaze directionof user 22, and the method ends.

In some embodiments, computer 26 extracts the 3D coordinates of head 32by identifying, from the 3D map, a position of the head along X-axis 40,Y-axis 42 and Z-axis 44. In alternative embodiments, computer 26extracts the 3D coordinates of head 32 by identifying, from the 2D imagea first position of the head along X-axis 40 and Y-axis 42, andidentifying, from the 3D map, a second position of the head along Z-axis44.

Embodiments of the present invention can use gaze detection tocontrolling a function of a computerized system such as computer 26responsively to the direction of the gaze. In some embodiments, computer26 can identify a given interactive item 36 presented, on display 28, ata position in the direction of the gaze, and change a state of the giveninteractive item responsively to the direction of the gaze.

In a first embodiment, changing the state of the given interactive item36 may comprise performing an operation associated with the giveninteractive item. For example, interactive items 36 may comprise menuchoices that user 22 can select to present specific content (e.g., amovie or a television show) on display 28. In a second embodiment,computer 26 can change the state of a given interactive item 36 bydirecting input received from the user to the given interactive item.For example, the given interactive item may comprise a text box, and ifthe user is gazing at the text box, computer 26 can direct anyalphanumeric input received from a keyboard to the text box.

In alternative embodiments, computer 26 can identify a given interactiveitem 36 presented, on display 28, at a position in the direction of thegaze, and change a state of the given interactive item responsively to avocal command received from the user. For example, the given interactiveitem may comprise an icon associated with a software application, andthe user can gaze at the icon and say the word “start” to execute theapplication. In additional embodiments, user interface 20 may beconfigured to identify a given interactive item 36 responsively to thedirection of the gaze, and to manipulate the given interactive itemresponsively to a gesture performed by a limb (e.g., finger 30 or hand31). For example, after selecting the given interactive item, if thecomputer receives a sequence of 3D maps indicating that the user ismoving hand 30 in a swiping motion (i.e., along a plane comprisingX-axis 40 and Y-axis 42), computer 26 can responsively reposition theselected interactive item in the direction of the swipe (e.g., left toright).

In further embodiments, embodiments of the present invention may be usedto receive an image (either a 2D image or a 3D map) including eye 34,identify, based on the image, a gaze direction, identify, based on thegaze direction a region on the display and in the direction of the gaze,and perform an operation on the region. For example, computer 26 maycomprise a tablet computer incorporating a digital camera, display 28may comprise a display for the tablet, and the camera may be configuredto focus the camera's lens on an item presented in the identifiedregion.

Additionally or alternatively, computer 26 can identify a device coupledto the computer and positioned in the direction of the gaze, andcontrolling a function of the device responsively to the direction ofthe gaze. For example, if the user gazes at a top of a speaker coupledto the computer, the computer can raise the volume level of the speaker,and if the user gazes at the bottom of the speaker, the computer canlower the volume level.

FIG. 4 is a schematic representation of a numeric keypad 90 presented ondisplay 28, configured for entering a password, in accordance with anembodiment of the present invention. The presence and gaze of user 22 inthis and subsequent figures is represented by eye 34, having a line ofsight 92. Sensing device 24 determines the line of sight by finding the3D location of the user's head and the gaze direction of eye 34 within afield of view 94 of the device, as described above.

In the description herein, interactive items 36 may be differentiated byusing a different description (e.g., icon or scroll box instead ofinteractive item) and appending a letter to the identifying numeral. Forexample, in FIG. 4, interactive items 46 comprise numeric input keys 36Aand a start button 36B. Upon presenting the input keys and the startbutton, computer 26 prompts user 22 to enter a password. Althoughnumeric keys are shown in FIG. 4, any sort of on-screen graphicalelements may be used as “keys” for this purpose. When the user wishes toinitiate operation of computer 26 or of a particular applicationexecuting on the computer, the user can direct his gaze from one givenkey 36A to the next in the proper sequence in order to enter a password.Optionally, the user may make a particular hand gesture or voice command(such as “ENTER”) as his gaze rests on each given key 36A (i.e., in asequence of keys comprising the password), in order to inform device 24to record the selection.

In some embodiments, computer 26 can be configured to select a sequenceof interactive items 36 (e.g., as the user moves his gaze from oneinteractive item to the next) with or without any additional input(i.e., a gesture or sound) made by the user to indicate which of theinteractive item has been selected. Similarly, computer 26 can beconfigured not to provide any visual or audio feedback to the userindicating the selection. As a result, even if the user enters apassword in a public place, onlookers will be unable to determine orcopy the password.

After entering the password, the user may direct his gaze toward startbutton 36B in order to continue interacting with the computer.

In addition to password entry, gaze detection can be used to enhancesecurity in other ways. For example, computer 26 may learn the user'scharacteristic eye movement patterns and/or other biometric features ofthe eyes 34 as an additional means of identification. As anotherexample, device 24 may be configured to find the gaze angle not only ofuser 22, but also of other people positioned within field of view 94 (asidentified by skeleton reconstruction from the 3D maps, for instance).In this case, device 24 may be configured to alert user 22 when anotherperson is looking at display 28 (and may even prompt computer 26 todisplay an image of this other person, captured by device 24, on thedisplay). This sort of functionality can assist user 22 in protectinghimself from eavesdropping and in deterring eavesdroppers fromcontinuing to look at the display.

As a power saving feature, device 24 may detect when the user is notlooking at display 28, and computer 26 may activate power savingtechniques when the user looks away for more than some threshold periodof time. For example, computer 26 may dim or darken the display (i.e.,decrease the brightness) entirely when the user is not looking at thedisplay. When the user looks back toward the display, the computer maydeactivate the power saving techniques. For example, computer 26 mayincrease the brightness to return it to full brightness upon detectingthat user 22 returned his gaze toward display. This sort ofgaze-dependent screen control is also useful in enhancing battery lifeof portable devices and reducing power consumption generally for costsaving and environmental friendliness.

Gaze Operated User Interface

Gaze tracking may be used to create an interactive user interface thatcan detect which on-screen interactive item the user is looking at (suchas a text box, or an application such as a word processor), therebyobviating the need for a mouse and/or a keyboard. For example, when theuser types text, the text is automatically directed to the text box atwhich the user is looking. As another example, when the user makes thefirst keystroke in a word or sentence, a “mouse click” type event issent to the looked-at text box, which causes the text to be typed intothe text box. In this manner the user may fill in a Web form without theneed to move the mouse to switch from field to field (“first name” to“last name” to “password,” etc.).

Furthermore, the combination of gaze tracking with other modalities,such as 3D mapping/gesture detection and/or voice detection, enables theuser to control on-screen objects fully, without the use of a mouse or atouch screen. In this manner, the user can perform a full range ofpointing and selection functions, including searching through largenumbers of information items and choices. The combined interfacemodalities may also be used to search and perform control functionswithin the context of a certain interactive item, such as performingfind, cut, copy and paste functions within an open file.

FIG. 5 is a flow diagram that schematically illustrates a method ofinteracting with a gaze operated user interface, in accordance with anembodiment of the present invention, and FIGS. 6A-6C are schematicrepresentations illustrating a sequence of operations performed using agaze operated user interface, in accordance with an embodiment of thepresent invention. In an initialization step 100, a computerized systemsuch as computer 26 presents multiple interactive items 36 on display28. In the configuration shown in FIGS. 6A-6C, computer 26 initiallypresents start button 36B in the lower right corner of display 28. Asuser 22 directs his line of sight 92 toward start button 36B, computer26 receives an input (e.g., depth maps) indicating a gaze direction ofuser 22 in a receive step 102. Upon determining the gaze direction,computer 26 identifies a target point 120 on display 28 based on thegaze direction (i.e., a point on the display that the user is lookingat) in an identification step 104.

In a comparison step 106, if a first interactive item 36 is in proximityto target point, then in an association step 108, computer 26 associatesthe target point with the first interactive item. In the example shownin FIG. 6A, the first interactive item comprises start button 36B, andtarget point 120 is positioned on the start button 36. Returning to step106, if the target point is not in proximity to any of the interactiveitems, then the method continues with step 102.

In a response step 110, computer 26 opens (i.e., presents) one or moresecond interactive items 36 in proximity to the first interactive item,and the method ends. Computer 26 can the select one of the secondinteractive items responsively to the user adjusting the gaze directionso that the target point is in proximity to the one of the secondinteractive items.

While the configuration of the interactive user interface in FIGS. 6A-6B36 show computer 26 presenting the second interactive items 36 radiallyoutward from the first interactive item in a concentric pattern, otherconfigurations are considered to be within the spirit and scope of thepresent invention. For example, computer 26 may present the first andsecond interactive items as nested rectangular icons.

In some embodiments, computer 26 can open the one or more secondinteractive items responsively to a gesture performed by a limb (e.g.,finger 30 or hand 31). For example, after associating the target pointwith the first interactive item, if the computer receives a sequence of3D maps indicating that the user is moving hand 30 in a swiping motion(i.e., along a plane comprising X-axis 40 and Y-axis 42), computer 26can responsively open the one or more second interactive items.

In the configuration shown in FIG. 6A, computer 26 presents a menu area122 of second interactive items comprising tabs 36C (arranged in aconcentric pattern) for various applications that are available on thecomputer. In some embodiments, computer 26 opens menu area 122 inresponse to the user gazing at target point 120, without necessarilyrequiring the user to make any sort of physical gesture or voicecommand.

In FIG. 6B, user 22 has directed his line of sight 92 toward a MAIL tab36C, thus causing computer 26 to automatically open the user'selectronic mail (e-mail) inbox, which contains listings 36D of incomingmail items. As the user scans his gaze up and down over listings 36D andbrings line of sight 92 to rest on a particular listing 36D, thecomputer automatically presents a preview 124 of the content of the mailitem represented by the listing. The user may preview other listings 36Dby moving his line of sight 92 up or down, or may return to the mainmenu by directing his line of sight 92 to a BACK button 36E. All ofthese actions may be performed by eye movement alone.

To open a selected mail item, the user may input a command to computer26 by another modality. For example, the user may say “OPEN,” or maymake an opening hand gesture while gazing at the listing 36Dcorresponding to the mail item. Sensing device 24 detects the audioinput or the 3D motion made by the user and inputs the appropriatecommand to computer 26. As a result, the screen shown in FIG. 6C opens,presenting full content 126 of the selected mail item. The user mayreturn to the inbox from this point by moving his gaze to a “BACK TOINBOX” area 36F, or may return to the main menu using BACK button 36E,as mentioned above.

When selecting a given interactive item 36, computer 26 can conveyvisual feedback to the user indicating the selection (i.e., beforeperforming an action such as presenting full content 126). Examples ofvisual feedback include changing the size and/or appearance of theselected item, or highlighting the selected item by surrounding theselected item with a border. Conveying visual feedback enables user 22to focus his gaze in the vicinity of the target point, thereby enhancingthe user experience. For example, when the user selects start button36B, computer 26 can convey visual feedback via the start button,thereby directing the user to maintain his gaze in proximity to wherethe computer presents tabs 36C in menu area 122.

Gaze Related Pointing Gestures

As described in detail hereinbelow, user 22 points to a giveninteractive item 36 while performing the Point-Select, Point-Touch andPoint-Context gestures. In embodiments of the present invention,computer 26 identifies where the user is pointing by defining a linebetween a first point and a second point of the user, and identifyingwhere (i.e., target point 120) the line intersects display 28.

FIG. 7 is a flow diagram that schematically illustrates a method ofdetecting a gaze related pointing gesture, in accordance with anembodiment of the present invention, FIG. 8 is a schematic pictorialillustration of user 22 performing a Point-Select gesture to select afirst given interactive item 36 presented on display 28, in accordancewith an embodiment of the present invention, and FIG. 9 is a schematicpictorial illustration of user 22 performing a Point-Touch gesture tomanipulate a second given interactive item 36 presented on display 28,in accordance with an embodiment of the present invention. Examples ofthe first interactive items that user 22 can select using thePoint-Select gesture include icons 36G, and examples of the secondinteractive items that user 22 can manipulate using the Point-Touchgesture include icons 36G, a vertical scroll box 36H and a horizontalscroll box 36I.

The Point-Select gesture comprises user 22 pointing a given finger 30(typically an index finger) toward the first given interactive item 36,moving finger 30 (typically along Z-axis 44) toward the first giveninteractive item, stopping or slowing down the finger, and then pullingthe finger back to the user. The Point-Touch gesture comprises user 22pointing finger toward a second given interactive item 36, moving finger30 (typically along Z-axis 44) toward the second given interactive item,pausing the finger, and then moving the hand along a plane comprisingX-axis 40 and Y-axis 42.

In a first receive step 130, a computerized system such as computer 26receives a first sequence of 3D maps that include user 22, and in anextraction step 132, the computer segments the received 3D maps toextract 3D coordinates of the first point and the second point of theuser. While FIGS. 8 and 9 show the first point comprising eye 34 and thesecond point comprising a tip of (index) finger 30, different firstpoints and second points are considered to be within the spirit andscope of the present invention. For example, the first point maycomprise a point between eyes 34, or some other point on the user'sface, and the second point may comprise any point on any of the user'slimbs. For different postures of user 22, computer 26 may use differentcalculations to identify the first and the second points.

In a first comparison step 134, if the first sequence of 3D mapsindicates that the user is moving the second point with respect to(typically toward) display 28, then in a second receive step 136,computer 26 receives and segments a second sequence of 3D mapsindicating a deceleration of the second point. In a calculation step138, computer 26 calculates a line segment 160 that intersects the firstpoint and the second point, and in an identification step 140, thecomputer extends line segment 160 to display 28, and identifies targetpoint 120 where line segment 160 intersects the display.

In some embodiments, user 22 may not be looking at the target point. Forexample, computer 26 may determine (i.e., using the gaze detectionembodiments described supra) that the user is directing his gaze at afirst given interactive item 36 that is presented on the left side ofdisplay 28, but step 140 identifies that the user is pointing to asecond given interactive 36 that is positioned on the right side ofdisplay 28. In this instance, the computer can be configured to selectthe second interactive item, even though the user is directing his gazetoward the first interactive item.

In a second comparison step 142, if computer 26 is presenting a giveninteractive item 36 in proximity to target point 120, then the computerengages the given interactive item 36 in proximity to the target point,and receives and segments a third sequence of 3D maps in a third receivestep 144. In a third comparison step 146, if the third sequence of 3Dmaps indicates that the user is moving the second point away from thedisplay (i.e., toward user 22), then in a selection step 148, computer26 engages (i.e., selects) the given interactive item presented inproximity to target point 120, and the method ends. In the example shownin FIG. 8, user 22 can select the icon “A” using the Point-Selectgesture comprising pointing finger 30 at the icon “A”, and pulling thefinger back.

In some embodiments engaging the given interactive item may compriseperforming an action associated with the given interactive item. Forexample, the given interactive item may comprise a given icon 36G for amovie, and engaging the given icon 36G comprises executing anapplication to view the movie.

Alternatively, while pointing at the icon “A”, user 22 can issue a vocalcommand such as “Select”. Upon receiving the vocal command frommicrophone 54, computer 26 can perform an operation associated with thegiven interactive item presented in proximity to target point 120 (e.g.,the icon “A”).

In some embodiments, computer 26 may select, using the Point-Selectgesture, a given interactive item 36, even if the gaze detectionembodiments described supra indicate that user is not looking at thegiven interactive item. In a first example, if computer 26 detects thatthe user has not changed his gaze direction for a specified time period,and computer 26 identifies a Point-Select gesture while the received 2Dimage and 3D map do not indicate a gaze direction (e.g., hand 31 isblocking eye 34), then computer 26 may “override” gaze detection andrespond to the Point-Select gesture.

In a second example, computer 26 may respond to a Point-Select gesture,if the finger motion gesture is “significant” (i.e., having at least afirst specified difference) and if target point 120 is at least a secondspecified distance from the interactive item presented in the directionof the gaze. For example, if the user is looking at a first interactiveitem 36 presented on a left side of display 28, and performs aPoint-Select gesture to a second interactive item 36 presented on anright side of the display, computer 26 may be configured to engage thesecond interactive item.

If the third sequence of 3D maps do not indicate that the user is movingthe second point away from display 28, then in a fourth comparison step150, computer 26 analyzes the third sequence of 3D maps to determine ifuser 22 is moving the second point along an X-Y plane comprising X-axis40 and Y-axis 42. If the third sequence of 3D maps indicates that theuser is moving the second point along the X-Y plane, then in arepositioning step 152, computer 26 repositions the interactive itempositioned in proximity to target point 120 responsively to the motionof the second point, and the method ends. In the example shown in FIG.9, user 22 can move vertical scroll box 36H using the Point-Touchgesture comprising pointing finger 30 at the vertical scroll box, andmoving the finger up or down.

If the third sequence of 3D maps do not indicate that the user is movingthe second point along the X-Y plane, then in a fifth comparison step154, computer 26 analyzes the third sequence of 3D maps to determine ifuser 22 is holding the second point relatively steady for at least aspecific time period (e.g., two second). If the third sequence of 3Dmaps indicates that the user is holding the second point relativelysteady for at least the specific time period, then in a presentationstep 156, computer 26 performs a hold operation on the for theinteractive item positioned in proximity to target point 120. Forexample, the hold operation may comprise presenting context informationfor the interactive item positioned in proximity to target point 120.Examples of context information include properties of the interactiveitem, or options available to the user. For example, computer 26 canpresent a message “Pull finger back to select, or move fingerhorizontally or vertically to reposition the item.” Another example ofthe hold operation may comprise computer 26 deleting the interactiveitem positioned in proximity to target point 120.

Returning to step 154, if the third sequence of 3D maps does notindicate that the user held the second point relatively steady for thespecific time period, then the method continues with step 130. Likewise,if computer 26 is not presenting a given interactive item 36 inproximity to target point 120 in step 142, or if the first sequence of3D maps does not indicate that user 22 is moving the second point towarddisplay 28 in step 134, then the method also continues with step 130.

In some embodiments, computer 26 can control a function of the devicecoupled to the computer in response to the gestures described herein.For example, if the user performs a Point-Touch gesture in the updirection while pointing toward a speaker coupled to the computer, thecomputer can raise the volume level of the speaker. Similarly, if theuser performs a Point-Touch gesture in the down direction while pointingtoward the speaker, the computer can lower the volume level of thespeaker.

While the gestures described in the flow diagram of FIG. 7 include thePoint-Select and the Point-Touch gestures, other pointing gestures thatidentify target point 120 via line segment 160 (i.e. intersecting thefirst and second points) are considered to be within the spirit andscope of the present invention. Additionally or alternatively, thePoint-Select and the Point-Touch gestures may be used in conjunctionwith the gaze detection embodiments described supra.

For example, computer 26 can be configured to select (i.e.,Point-Select) a given interactive item 36 upon identifying targetposition 120 based on the user's gaze, the first sequence of 3D mapsindicating user 28 is moving finger 30 toward display 28, the secondsequence of 3D maps indicating that the user is decelerating the finger,and the third sequence of 3D maps indicating that the user is movingfinger 30 away from the display. Similarly, computer 26 can beconfigured to responsively reposition (i.e., Point-Touch) a giveninteractive item 36 upon identifying target position 120 based on theuser's gaze, the first sequence of 3D maps indicating user 28 is movingfinger 30 toward display 28, the second sequence of 3D maps indicatingthat the user is decelerating the finger, and the third sequence of 3Dmaps indicating that the user is moving finger 30 along the planecomprising X-axis 40 and Y-axis 42.

FIG. 10 is a schematic pictorial illustration showing an alternativePoint-Select gesture, also referred to herein as a Trigger gesture inaccordance with an embodiment of the present invention. After pointingindex finger 30 at a given interactive item 36, user 22 can select thegiven interactive item by raising or folding a thumb 170 (as indicatedby an arrow 172), where folding the thumb completes the gestures (i.e.,instead of moving the finger toward the given interactive item).Alternatively, computer 26 can select the given icon upon user 22pointing finger 30 at the given interactive item, and issuing an audiocommand to select the given interactive item (e.g., the user utters theword “open”).

Additionally or alternatively, while the example shown in FIGS. 8-9 showthe first point positioned on head 32, other first points are consideredto be within the spirit and scope of the present invention. In theexample shown in FIG. 10, the first point comprises a knuckle of indexfinger 30, and line segment 160 intersects the fingertip and theknuckle.

Non-Tactile User Interface Calibration

In operation, non-tactile user interface 20 is typically used by morethan one user, and each user may point differently at the same giveninteractive item 36 presented on display 28. In some embodiments, usingelements of the non-tactile user interface (e.g., icons 30), computer 26can calculate and store a calibration coefficient for each user of thenon-tactile user interface.

FIG. 11 is a schematic pictorial illustration of user 22 pointing finger30 at a given icon 36G, in accordance with an embodiment of the presentinvention. As shown in the Figure, based the user's positioning offinger 30, line segment 160 points just below the icon “A”. Therefore,upon identifying user 22, computer 26 can apply an appropriatecalibration coefficient when detecting gestures performed by the user.

For example, if non-tactile user interface 20 is inactive for aspecified period of time, computer 26 may “lock” the non-tactile userinterface, and present a single unlock icon 36G that user 22 can pointto in order to unlock the user interface. As the user points at theunlock icon, computer 26 can identify target point 120 and calculate acalibration coefficient for the user (i.e., based on a proximity of thetarget point to the unlock icon).

Additionally or alternatively, there may be instances when user 22performs a gesture, target point 120 is located between two or moreicons 36G, and computer cannot identify which of the icons the user ispointing to. In instances when computer 26 cannot identify which icon36G is being pointed to, the computer can identify a subset of the iconsthat are in proximity to target point 120, present the identified iconsin a larger size, and prompt the user to point again.

FIG. 12A is a schematic pictorial illustration of user 22 positioningfinger 30 to point at one of icons 36G, in accordance with an embodimentof the present invention. As shown in the figure, target point 120 islocated between the icons “A”, “B”, “E”, and “F”.

FIG. 12B is a schematic pictorial illustration of user 22 positioningfinger 30 to point at a given icon 36G, where computer 26 presents theicons in a larger size, in accordance with an embodiment of the presentinvention. Continuing the example shown in FIG. 12A, computer 26presents the icons “A”, “B”, “E”, and “F” in a larger size, and user 22positions finger 30 to point at the icon “A”.

In a similar manner, user interface 20 can calibrate user 22, when theuser selects a given icon 36G via gaze detection derived from a 2D imageand/or a 3D map, as described supra. In a first, computer 26 may “lock”the non-tactile user interface, and present a single unlock icon 36Gthat user 22 can look at to in order to unlock the user interface. Asthe user gazes at the unlock icon, computer 26 can identify target point120 and calculate a calibration coefficient for the user (i.e., based ona proximity of the target point to the unlock icon).

In a second example, if the user is gazing at a given interactive item36G for a specific period of time, computer 26 may present contextinformation for the given interactive item in an interactive item 36(e.g., a pop-up dialog box). If the user gazes in proximity to theinteractive item presenting the context information, computer 26 cancalculate a calibration coefficient based on the proximity of the targetpoint to the context information.

Middleware

As described supra, middleware for extracting higher-level informationfrom the 3D maps and gaze information may run on processor 50 and/or CPU68, and CPU 68 may execute application programs which drive userinterface 72 based on information provided by the middleware, typicallyvia an API.

The following are examples of middleware primitives that computer 26 canbe use to extract information from 3D maps received from device 24:

-   -   InteractStart( ): Identifies a beginning of an interaction with        user interface 20.    -   InteractHover(Pos2D, Radius): Identifies current target point        120 on display 28 (i.e., coordinates on the display where user        22 is pointing finger 30). The Pos2D parameter references a (two        dimensional) location on display 28.    -   InteractPointNew(Pos2D, Radius): Identifies target point 120        when user 22 performs a Point-Select gesture.    -   InteractPointUpdate(Pos2D, Radius): Updates target point 120 as        user 22 moves finger 30 along the X-Y plane while performing the        Point-Touch gesture.    -   InteractEnd(Pos2D, Radius): Identifies when user 22 moves finger        30 outside field of view 94.

Using the middleware functions described supra, computer 26 can identifythe following “stages” as user 22 performs the Point-Select and thePoint-Touch gestures:

-   -   1. Finger Identification. During the Finger Identification        stage, the middleware identifies and starts tracking finger 30.        The middleware tracks the finger, and identifies any intent in        the finger's motion toward the display. This enables a user        interface 20 to differentiate a pointing gesture from other        casual hand movements.    -   2. Gesture Identification. To reach the Gesture Identification        stage, two conditions have typically been met: (a) User 22 moved        finger 30 toward display 28, and (b) Target point 120 is within        the boundaries of the display. In some configurations, target        point 120 can be “located” slightly outside display 28. For        example, since a valid Point-Touch gesture may include dragging        a given interactive item 36 from “outside” display 28 (e.g.,        backspace, alt-tab etc.).    -   Upon entering the second stage, the middleware conveys an        InteractStart( ) event. Upon conveying the InteractStart( )        event, the middleware is tracking finger 30 constantly and        looking for a change in direction which defines the target point        (see stage 3 below). While the finger is moving, the middleware        is conveying InteractHover(Pos2D, Radius) events, which enables        user interface 20 to detect where user 22 is currently pointing        finger 30.    -   3. Interaction Point. When user 20 stops finger 30, or moves        finger 30 away from display 28, the Interaction Point stage is        reached. Based on a location of the second point, the middleware        calculates target point 120 by connecting (for example) eyes 30        and the finger point to create line segment 160, and extends the        line segment to reach the display. The middleware then conveys        the event InteractPointNew(Pos2D, Radius) which identifies        target point 120, thereby enabling user interface 20 to select        the intended interactive item 36.    -   4. Interaction Ends. User 22 moving finger 30 away from display        28 and outside field of view 94 indicates completion of a        gesture (e.g., the Point-Select and the Point-Touch gestures        described supra), thereby disengaging the user from user        interface 20. To re-engage user interface 20, user 22 can        reposition finger 30 within field of view 94, thereby entering        the finger identified stage described hereinabove.

At a higher level of abstraction, the middleware primitives describedsupra can be combined to define the following primitives that aresimilar to existing touch screen primitives that are known in the art:

-   -   Activate: The Point-Select gesture that engages (i.e.,        “clicking”) a given interactive item 36. The Activate primitive        can be used to activate an application, pressing button, or        follow a hyperlink.    -   Pan: The Point-Touch gesture that moves a given interactive item        36 in any direction on the X-Y plane. In operation, some        applications may be configured to react to movements on either        the X-axis or the Y-axis.    -   Context: As described supra, the Point-Hold gesture comprising        user 22 moving finger 30 toward a given interactive item 36, and        the holding the finger relatively steady for a specific timer        period (i.e., similar to positioning a mouse over an Item and        pressing the right mouse button in a Microsoft Windows™        environment). In response to the Point-Hold gesture, computer 26        can convey feedback to the user indicating what to do next        (e.g., dragging and/or drop a given icon 36G), or information on        the given interactive item (e.g., movie plot summary as        described supra).

In operation, the following events can be conveyed by the middleware inorder to allow applications executing on computer 26 to be developed ata higher abstraction level which reflects user experience (UX) languageprimitives:

-   -   Start(Pos2D, Radius): Initiate interaction with user interface        20 in a manner similar to the InteractStart( ) event but at a        higher level of abstraction.    -   Activate(Pos2D, Radius): Activate one of the gesture primitives        described hereinbelow.    -   Pan(Pos2D, radius): User 22 initiates a Point-Touch gesture.    -   PanEnd(pos2D, Radius): User 22 completes (i.e., disengages from)        a Point-Touch pan gesture (e.g., by moving finger 30 back away        from the display).    -   Context(Point2D, Radius): User 22 initiates a Point-Hold        gesture. In operation, user 22 may transition directly from the        Point-Hold gesture to the Point-Touch gesture.    -   ContextEnd( ): User 22 disengages from user interface 20 upon        completing the Point-Hold gesture (i.e., without transitioning        to the Point-Touch gesture).

Interactive 3D Displays and Games

The combined 3D mapping and gaze direction information provided bysensing device 24 may be used in various ways to enhance the quality anduser experience of 3D graphical rendering and 3D image presentation. Forexample, in an interactive game in which the user moves through a scene,scenery on display 28 may be brought into focus in the direction of theline of sight 92. Scenery in other areas (such as distant scenery whenthe user's point of view is on a nearby item, or vice versa) may beintentionally blurred, in order to simulate actual depth accommodationand/or to save bandwidth.

As another example, a game application running on computer 26 may beprogrammed to change a “story line” of the game depending on the user'sgaze direction. For instance, the computer may surprise the user bypresenting items (such as “enemy” characters) suddenly in areas of thedisplay where the user is not looking at a given moment. The methods ofgaze-directed pointing and selection that were described above inreference to FIGS. 6A-6C may likewise be applied in selectinginteractive items 36 and “aiming” them (such as pointing a weapon at atarget) in games and other “virtual worlds.”

Depth and gaze information collected by device 24 can be used inenhancing the capabilities and user experience of 3D displays,particularly autostereoscopic displays. Such displays operate bypresenting different images to the user's right and left eyes, butgenerally can be viewed only from a limited range of positions. Bytracking the user's head location and gaze direction, device 24 may beable to direct the display to modify any images it presents so that theycan be viewed over a larger range of positions and show differentangular views of the items presented on the display. The parallaxapplied to near-field items 36 that are presented on theautostereoscopic display (or other 3D display) can be modified dependingon the distance of the user's head from the display, in order to enhancerealism and reduce visual discomfort that some users may feel in thisenvironment.

This sort of 3D display can also be driven to interact with the user's3D gestures. For example, based on a known location of the user, as wellas the user's gaze direction, computer 26 can drive the 3D display todisplay virtual items in space at locations where the user can “touch”them. The user can then manipulate and interact with the virtual itemsby moving his hands (or other body parts) in the locations of the itemsin 3D space. Device 24 senses the user's gestures and providesappropriate input to computer 26, so that the computer can move orotherwise modify the items in response to the user interactions. Thissort of interaction model also enables the user to reach for andinteract with a given on-screen object that is located “behind” anotherobject in the virtual space created by the display.

In an additional example, computer 26 may present content comprisingmultiple interactive items 36 (e.g., characters in a game) on display28. In some embodiments, computer 26 can identify a region (i.e., anarea around target point 12) on the display in the direction of theuser's gaze, and present the content within the identified region as “infocus” (i.e., present clearly), and the content outside the identifiedregion as “out of focus” (i.e. present blurred). In an alternativeembodiment, each of the multiple interactive items may have anassociated depth value, and as the user gazes toward a given interactiveitem, the computer can simulate a 3D environment by presenting theinteractive items whose associated depth values are in accordance withthe depth value if the given interactive item as “in focus” (i.e.,present clearly), and presenting the interactive items whose associateddepth values are not in accordance with the depth value if the giveninteractive item as “out of focus” (i.e. present blurred).

In a further example, while playing a game, user 22 can select a weaponby pointing at a weapon presented as an interactive icon on the display.If the selected weapon is a gun, the user can “aim” the gun using thePoint-Touch gesture, and “shoot” the gun using the Point-Select gestureor the Trigger gestures described supra. Alternatively, if the selectedweapon is a sword, then the user can manipulate the sword in threedimensions (i.e., along the X-Y plane and the Z-axis) by using acombination of the Point-Select and the Point-Touch gestures.

The above applications are just a few examples of how mixed modalityuser interfaces can be used to enhance system capabilities and userexperience, and other, similar applications are considered to be withinthe scope of the present invention. As another example, the capabilitiesof sensing device 24 may be used in gauging user interest in contentsuch as a Web site or video program, depending on the on-screen objecton which the user fixed his gaze, as well as whether the user waslooking at the display at all while a certain program (such as acommercial) was presented. These capabilities may similarly be used inextracting user interest profiles, as described, for example, in U.S.patent application Ser. No. 13/295,106, filed Nov. 14, 2011, which isincorporated herein by reference.

It will be appreciated that the embodiments described above are cited byway of example, and that the present invention is not limited to whathas been particularly shown and described hereinabove. Rather, the scopeof the present invention includes both combinations and subcombinationsof the various features described hereinabove, as well as variations andmodifications thereof which would occur to persons skilled in the artupon reading the foregoing description and which are not disclosed inthe prior art.

The invention claimed is:
 1. An apparatus, comprising: at least onesensing device configured to produce a sequence of three dimensional(3D) maps of at least a part of a body of a user of a computerizedsystem and to detect light reflected off an element of an eye of theuser; and a processor coupled to the at least one sensing device andconfigured to process the 3D maps in order to identify a gestureperformed by the user and to identify a direction of a gaze of the userbased on the light reflected off the element of the eye, and to controla function of the computerized system responsively to the gesture andthe direction of the gaze.
 2. The apparatus according to claim 1,wherein the element is selected from a list comprising a pupil, an irisand a cornea.
 3. The apparatus according to claim 1, wherein theprocessor is configured to identify the gesture responsively to aposition of a finger in the 3D maps.
 4. The apparatus according to claim1, wherein controlling the function of the computerized system comprisesidentifying an interactive item presented on a display coupled to thecomputerized system and in the direction of the gaze, and changing astate of the interactive item.
 5. The apparatus according to claim 1,wherein controlling the function of the computerized system comprisesidentifying a device coupled to the computerized system and in thedirection of the gaze, and changing a state of the device.
 6. A computersoftware product comprising a non-transitory computer-readable medium,in which program instructions are stored, which instructions, when readby a computer, cause the computer to receive a sequence ofthree-dimensional (3D) maps of at least a part of a body of a user ofthe computer, and to receive an input indicative of light reflected offan element of an eye of the user, wherein the instructions cause thecomputer to process the 3D maps in order to identify a gesture performedby the user and to identify a direction of a gaze of the user based onthe light reflected off the element of the eye, and to control afunction of the computerized system responsively to the gesture and thedirection of the gaze.
 7. A method, comprising: receiving a sequence ofthree-dimensional (3D) maps of at least a part of a body of a user of acomputerized system; processing the 3D maps in order to identify agesture performed by the user; identifying a direction of a gaze of theuser by analyzing light reflected off an element of the eye; andcontrolling a function of the computerized system responsively to thegesture and the direction of the gaze.
 8. The method according to claim7, wherein the element is selected from a list comprising a pupil, aniris and a cornea.
 9. The method according to claim 7, whereinprocessing the 3D maps comprises identifying a position of a finger inthe 3D maps.
 10. The method according to claim 9, wherein identifyingthe position of the finger comprises detecting a point-select gesturewith respect to an interactive item, in which the finger moves towardthe interactive item, and then moves along a plane from the interactiveitem.
 11. The method according to claim 9, wherein identifying theposition of the finger comprises detecting a point-touch gesture withrespect to an interactive item, in which the finger makes an initialmovement toward the interactive item, and then moves in a directionperpendicular to the initial movement.
 12. The method according to claim9, wherein identifying the position of the finger comprises detectingrespective locations of a tip and a knuckle of the finger, andidentifying a direction in which the finger is pointing responsively tothe detected locations.
 13. The method according to claim 7, whereincontrolling the function of the computerized system comprisesidentifying an interactive item presented on a display coupled to thecomputerized system and in the direction of the gaze, and changing astate of the interactive item.
 14. The method according to claim 13,wherein the state of the interactive item is changed responsively to thegaze.
 15. The method according to claim 13, wherein the state of theinteractive item is changed responsively to a vocal command receivedfrom the user.
 16. The method according to claim 13, wherein the stateof the interactive item is changed responsively to the gesture.
 17. Themethod according to claim 13, wherein identifying the position of thefinger comprises detecting a pointing motion of a finger, and whereinchanging the state comprises moving the identified interactive item onthe display in a direction pointed to by the finger.
 18. The methodaccording to claim 7, wherein controlling the function of thecomputerized system comprises performing an action associated with aninteractive item presented in the direction of the gaze on a displaycoupled to the computerized system, upon detecting that the gesturecomprises a motion of a limb toward the display, followed by adeceleration of the motion of the limb toward the display, and thenfollowed by a motion of the limb away from the display.
 19. The methodaccording to claim 7, wherein controlling the function of thecomputerized system comprises activating, by the computerized system, apower saving technique upon the user directing the gaze away from thedisplay.
 20. The method according to claim 7, wherein controlling thefunction of the computerized system comprises identifying a devicecoupled to the computerized system and in the direction of the gaze, andchanging a state of the device.